The goal of this project is to detect lane lines on the road using a video feed. The code will analyze each image frame in the feed and use computer vision techniques to detect the location of the lane lines.
A few assumptions were taken when putting the project together:
The code used is Python 3.5 and the following techniques are used for line detection:
For further code documentation or reference on reproducing the project, please refer to the README.
In [1]:
import matplotlib.pyplot as plt
import matplotlib.image as mpimg
import numpy as np
import math
import cv2
import glob
%matplotlib inline
from collections import deque
In [2]:
class ImageHandler:
def read(self, directory):
return [plt.imread(path) for path in glob.glob(directory)]
def save(self, directory, imgs, suffix="", cmap=None):
for i, img in enumerate(imgs):
cmap = 'gray' if len(img.shape)==2 else cmap
plt.imshow(img, cmap=cmap)
plt.xticks([])
plt.yticks([])
plt.tight_layout(pad=0, h_pad=0, w_pad=0)
plt.savefig(directory+"Image"+str(i)+suffix+".jpg")
plt.close()
return "done"
def show(self, imgs, cmap=None):
rows = (len(imgs)+1)//2
plt.figure(figsize=(10, 11))
for i, img in enumerate(imgs):
plt.subplot(rows, 2, i+1)
cmap = 'gray' if len(img.shape)==2 else cmap
plt.imshow(img, cmap=cmap)
plt.xticks([])
plt.yticks([])
plt.tight_layout(pad=0, h_pad=0, w_pad=8)
plt.show()
def process(self, imgs, function):
return list(map(lambda img: function(img), imgs))
def process_and_append(self, set1, set2, function):
final = []
for set1_item, set2_item in zip(set1, set2):
final.append(function(set1_item, set2_item))
return final
Let's have a look at the test images:
In [3]:
ih = ImageHandler()
imgs = ih.read('test_images/*.jpg')
ih.show(imgs)
In this project we will address detecting the following types of lines: solid yellow, solid white and dotted white. Also, sunny clear images will be tested. It may require more complicated processing for night time or poor weather conditions.
In [4]:
class ImageProcessor:
def rgb_to_hsl(self, img):
return cv2.cvtColor(img, cv2.COLOR_RGB2HLS)
def rgb_to_hsv(self, img):
return cv2.cvtColor(img, cv2.COLOR_RGB2HSV)
#This appears to work the best when road hue changes (example: going over a bridge)
def hsl_to_white_yellow(self, img):
hsl = self.rgb_to_hsl(img)
white = cv2.inRange(hsl, np.uint8([ 0 , 200 , 0 ]), np.uint8([ 255 , 255 , 255 ] ))
yellow = cv2.inRange(hsl, np.uint8([ 13 , 0 , 89 ]), np.uint8([ 55, 255 , 255 ]))
mask = cv2.bitwise_or(white, yellow)
return cv2.bitwise_and(img, img, mask = mask)
def hsv_to_white_yellow(self, img):
hsv = self.rgb_to_hsv(img)
white = cv2.inRange(hsv, np.uint8([ 0 , 0 , 220 ]), np.uint8([ 255 , 255 , 255 ]))
yellow = cv2.inRange(hsv, np.uint8([ 13 , 10 , 255 ]), np.uint8([ 55 , 255 , 255 ]))
mask = cv2.bitwise_or(white, yellow)
return cv2.bitwise_and(img, img, mask = mask)
def to_grayscale(self, img):
return cv2.cvtColor(img, cv2.COLOR_RGB2GRAY)
def canny(self, img, low_threshold=55, high_threshold=144):
return cv2.Canny(img, low_threshold, high_threshold)
def gaussian_blur(self, img, kernel_size=13):
return cv2.GaussianBlur(img, (kernel_size, kernel_size), 0)
def region_of_interest(self, img):
"""
Applies an image mask.
Only keeps the region of the image defined by the polygon
formed from `vertices`. The rest of the image is set to black.
"""
#defining a blank mask to start with
mask = np.zeros_like(img)
#defining a 3 channel or 1 channel color to fill the mask with depending on the input image
if len(img.shape) > 2:
channel_count = img.shape[2] # i.e. 3 or 4 depending on your image
ignore_mask_color = (255,) * channel_count
else:
ignore_mask_color = 255
#filling pixels inside the polygon defined by "vertices" with the fill color
cv2.fillPoly(mask, self.get_vertices(img), ignore_mask_color)
#returning the image only where mask pixels are nonzero
masked_image = cv2.bitwise_and(img, mask)
return masked_image
def get_vertices(self, img):
imshape = img.shape
return np.array([[(0,imshape[0]+55),(imshape[1]/2-75, imshape[0]/2+70), (imshape[1]/2+75, imshape[0]/2+70), (imshape[1],imshape[0])]], dtype=np.int32)
def weighted_img(self, img, initial_img, α=1., β=0.8, λ=0.):
return cv2.addWeighted(initial_img, α, img, β, λ)
Documentation on this color space can be found here: HSL and HSV
In [5]:
ip = ImageProcessor()
imgs_hsl = ih.process(imgs, ip.rgb_to_hsl)
ih.save('test_images_out/', imgs_hsl, "_hsl")
ih.show(imgs_hsl)
I built the function hsl_to_white_yellow function to extract only white and yellow colors from the hls mask. This will help with isolating the lane lines and cleaning out any color deviations in the road. The following OpenCV functions are used:
I spent some time tuning the parameters in filtering out colors. This proves to have a reasonable significance on the quality of the final processing.
In [6]:
imgs_white_yellow = ih.process(imgs, ip.hsl_to_white_yellow)
#save for writeup
ih.save('test_images_out/', imgs_white_yellow, "_white_yellow")
ih.show(imgs_white_yellow)
It can be seen that the yellow and white lines are clearly shown after processing.
Next we will use canny edge detection to detect the line edges. The OpenCV functions used in this process will include the following:
See wikipedia article on Canny edge detection: Canny edge detector
In [7]:
edges = ih.process(ih.process(ih.process(imgs_white_yellow, ip.to_grayscale), ip.gaussian_blur), ip.canny)
ih.save('test_images_out/', edges, "_edges")
ih.show(edges)
Now that we have the edges, we will mask out any regions of the images that don't matter. This includes anything above the horizon or far sides of the image. Doing this will allow us to focus in on only the lane lines.
The OpenCV functions used in this process are:
In [8]:
roi_imgs = ih.process(edges, ip.region_of_interest)
ih.save('test_images_out/', roi_imgs, "_roi")
ih.show(roi_imgs)
We have successfully detected only the raw lane lines from the image
The following class will have the OpenCV Hough Line functionality. It will also contain functionality to draw pre-defined lines on an image.
Documentation on Hough lines can be found here: Hough transform
In [9]:
class LineDetector:
def get_hough_lines(self, img, rho=1, theta=np.pi/180, threshold=21, min_line_len=21, max_line_gap=233):
return cv2.HoughLinesP(img, rho, theta, threshold, np.array([]), minLineLength=min_line_len, maxLineGap=max_line_gap)
def hough_lines(self, img):
line_img = np.zeros((img.shape[0], img.shape[1], 3), dtype=np.uint8)
self.draw_lines(line_img, self.get_hough_lines(img))
return line_img
def draw_lines(self, img, lines, color=[255, 0, 0], thickness=2):
for line in lines:
for x1,y1,x2,y2 in line:
cv2.line(img, (x1,y1), (x2,y2), color, thickness)
return img
The following function will take the edges detected in the Canny edge detection algorithm and detect the actual lines that will be stored as vertex points. This step is crucial in taking raw data from the image files and converting it to mathematically modeled lines that can be used in computer algorithms.
In [10]:
ld = LineDetector()
line_imgs = ih.process(roi_imgs, ld.hough_lines)
ih.show(line_imgs)
Now that we have the lines, lets draw them back onto the original image for viewing purposes
In [11]:
imgs2 = np.copy(imgs)
roi_hough_lines = ih.process(roi_imgs, ld.get_hough_lines)
imgs_w_lines = ih.process_and_append(imgs2, roi_hough_lines, ld.draw_lines)
ih.save('test_images_out/', imgs_w_lines, "_lines")
ih.show(imgs_w_lines)
In [12]:
class LineProcessor:
"""
the below formulas calculate a linear regression on the set of hough lines
it gives equal weight to each line detected from the hough line function
this proved to not be as robust due to smaller lines causing increase fluctuation in the fitted line slope
simply selecting the largest detected hough line give a more accurate representation of the physical lane lines
"""
def get_linear_fit(self, x_set, y_set): #not currently used
X = np.array(x_set)
Y = np.array(y_set)
XdotX = X.dot(X)
Xmean = X.mean()
Xsum = X.sum()
XdotY = X.dot(Y)
Ymean = Y.mean()
denominator = XdotX - Xmean * Xsum
a = ( XdotY - Ymean * Xsum ) / denominator
b = ( Ymean * XdotX - Xmean * XdotY ) / denominator
return a,b
def get_left_right(self, lines):
left = None
right = None
left_line_size = 0
right_line_size = 0
for line in lines:
for x1, y1, x2, y2 in line:
if x2==x1:
continue # igonore vertical
a = (y2-y1)/(x2-x1)
b = y1-a*x1
length = np.sqrt((y2-y1)**2+(x2-x1)**2)
if a < 0: # left lane detected
if length > left_line_size:
left_line_size = length
left = a,b
else: # right lane detected
if length > right_line_size:
right_line_size = length
right = a,b
return left,right
def get_line_verticies(self, line):
if type(line) is np.ndarray:
line = line.tolist()[0]
half = int(len(line)/2)
return line[:half], line[half:]
def lines_to_verticies(self, lines):
verticies = []
for line in lines:
for vertex in self.get_line_verticies(line):
verticies.append(vertex)
verticies.sort()
return np.array([verticies])
def build_line(self, y1, y2, formula):
if formula is not None:
a, b = formula
if a != 0:
return [int((y1-b)/a), int(y1), int((y2-b)/a), int(y2)]
else:
return None
else:
return None
"""
This function takes a list of lines and finds the average of all the lines.
"""
def find_centerline(self, lines):
if len(lines)>0:
return np.mean(lines, axis=0, dtype=np.int32)
else:
return None
In [13]:
class LaneDetector:
def __init__(self):
self.LENGTH = 13 #Build a simple moving average algorithm for smoothing of detected lines
self.ip = ImageProcessor()
self.ld = LineDetector()
self.lp = LineProcessor()
self.left_lines = deque(maxlen=self.LENGTH)
self.right_lines = deque(maxlen=self.LENGTH)
def get_edges(self, img):
mask = ip.hsl_to_white_yellow(img)
gray = self.ip.to_grayscale(mask)
blurred = self.ip.gaussian_blur(gray)
return self.ip.canny(blurred)
def get_raw_lane_lines(self, img):
edges = self.get_edges(img)
roi = self.ip.region_of_interest(edges)
return self.ld.get_hough_lines(roi)
def get_lane_lines(self, img):
y1 = img.shape[0]
y2 = y1*0.618
lines = self.get_raw_lane_lines(img)
left_formula, right_formula = self.lp.get_left_right(lines)
return [self.lp.build_line(y1, y2, left_formula), self.lp.build_line(y1, y2, right_formula)]
def get_lane_lines_smooth(self, img):
[left_line, right_line] = self.get_lane_lines(img)
right_smooth = self.apply_smoothing(right_line, self.right_lines)
left_smooth = self.apply_smoothing(left_line, self.left_lines)
return [left_smooth.tolist() if left_smooth is not None else None, right_smooth.tolist() if right_smooth is not None else None]
def get_centerline(self, img):
lines = self.get_lane_lines_smooth(img)
centerline = self.lp.find_centerline(lines)
return [centerline]
def draw_lane_lines(self, img, lines, color=[255, 0, 0], thickness=13):
line_img = np.zeros_like(img)
self.ld.draw_lines(line_img, [lines], color, thickness)
return self.ip.weighted_img(line_img, img)
def draw_raw_lines(self, img, lines, color=[255, 0, 0], thickness=3):
line_img = np.zeros_like(img)
self.ld.draw_lines(line_img, lines, color, thickness)
return self.ip.weighted_img(line_img, img, 0.89)
def detect_centerline(self, img):
return self.draw_lane_lines(img, self.get_centerline(img))
def detect_lane_lines(self, img):
return self.draw_lane_lines(img, self.get_lane_lines(img))
def detect_lane_lines_smooth(self, img):
return self.draw_lane_lines(img, self.get_lane_lines_smooth(img))
def detect_raw_lane_lines(self, img):
return self.draw_raw_lines(img, self.get_raw_lane_lines(img))
def apply_smoothing(self, line, lines):
if line is not None:
lines.append(line)
return self.lp.find_centerline(lines)
def get_line_queue(self):
return self.left_lines, self.right_lines
def detect_lane_zone_raw(self, img):
zone = np.zeros_like(img)
cv2.fillPoly(zone, self.lp.lines_to_verticies(self.get_raw_lane_lines(img)), [0 , 255, 0])
return self.ip.weighted_img(img, zone, α=0.34)
def detect_lane_zone(self, img):
zone = np.zeros_like(img)
cv2.fillPoly(zone, self.lp.lines_to_verticies(self.get_lane_lines(img)), [0 , 255, 0])
return self.ip.weighted_img(img, zone, α=0.34)
def detect_lane_zone_smooth(self, img):
zone = np.zeros_like(img)
cv2.fillPoly(zone, self.lp.lines_to_verticies(self.get_lane_lines_smooth(img)), [0 , 255, 0])
return self.ip.weighted_img(img, zone, α=0.34)
Now we will used the Lane Detector class to detect and extrapolate lane lines on the original images
In [14]:
detector = LaneDetector()
detected_lanes = ih.process(imgs, detector.detect_lane_lines)
ih.save('test_images_out/', detected_lanes, "_lanes")
ih.show(detected_lanes)
The following items were not specified as requirements in the project but I decided to pursue them for further learning and understanding.
In [15]:
raw_zone_imgs = ih.process(imgs, detector.detect_lane_zone_raw)
ih.show(raw_zone_imgs)
In [16]:
zone_imgs = ih.process(imgs, detector.detect_lane_zone)
ih.save('test_images_out/', zone_imgs, "_zones")
ih.show(zone_imgs)
In [17]:
centerline_imgs = ih.process(imgs, detector.detect_centerline)
ih.show(centerline_imgs)
In [18]:
# Import everything needed to edit/save/watch video clips
from moviepy.editor import VideoFileClip
from IPython.display import HTML
In [19]:
class VideoProcessor:
detector = LaneDetector()
def save_to_file(self, video_input, video_output, function):
self.detector = LaneDetector()
clip1 = VideoFileClip(video_input)
clip = clip1.fl_image(function) #NOTE: this function expects color images!!
%time clip.write_videofile(video_output, audio=False)
clip1.reader.close()
def detect_lane_lines(self, img):
return self.detector.detect_lane_lines(img)
def detect_smooth_lane_lines(self, img):
return self.detector.detect_lane_lines_smooth(img)
def detect_raw_lane_lines(self, img):
return self.detector.detect_raw_lane_lines(img)
def detect_lane_zone(self, img):
return self.detector.detect_lane_zone(img)
def detect_lane_zone_raw(self, img):
return self.detector.detect_lane_zone_raw(img)
def detect_lane_zone_smooth(self, img):
return self.detector.detect_lane_zone_smooth(img)
def detect_centerline(self, img):
return self.detector.detect_centerline(img)
def detect_edges(self, img):
return self.detector.get_edges(img)
In [20]:
vp = VideoProcessor()
In [21]:
vp.save_to_file("test_videos/solidWhiteRight.mp4", "test_video_output/solidWhiteRight_raw.mp4", vp.detect_raw_lane_lines)
In [22]:
HTML("""
<video width="960" height="540" controls>
<source src="{0}">
</video>
""".format("test_video_output/solidWhiteRight_raw.mp4"))
Out[22]:
In [23]:
vp.save_to_file("test_videos/solidYellowLeft.mp4", "test_video_output/solidYellowLeft_raw.mp4", vp.detect_raw_lane_lines)
In [24]:
HTML("""
<video width="960" height="540" controls>
<source src="{0}">
</video>
""".format("test_video_output/solidYellowLeft_raw.mp4"))
Out[24]:
In [25]:
vp.save_to_file("test_videos/challenge.mp4", "test_video_output/challenge_raw.mp4", vp.detect_raw_lane_lines)
In [26]:
HTML("""
<video width="960" height="540" controls>
<source src="{0}">
</video>
""".format("test_video_output/challenge_raw.mp4"))
Out[26]:
In [27]:
vp.save_to_file("test_videos/solidWhiteRight.mp4", "test_video_output/solidWhiteRight_fit.mp4", vp.detect_lane_lines)
In [28]:
HTML("""
<video width="960" height="540" controls>
<source src="{0}">
</video>
""".format("test_video_output/solidWhiteRight_fit.mp4"))
Out[28]:
In [29]:
vp.save_to_file("test_videos/solidYellowLeft.mp4", "test_video_output/solidYellowLeft_fit.mp4", vp.detect_lane_lines)
In [30]:
HTML("""
<video width="960" height="540" controls>
<source src="{0}">
</video>
""".format("test_video_output/solidYellowLeft_fit.mp4"))
Out[30]:
In [31]:
vp.save_to_file("test_videos/challenge.mp4", "test_video_output/challenge_fit.mp4", vp.detect_lane_lines)
In [32]:
HTML("""
<video width="960" height="540" controls>
<source src="{0}">
</video>
""".format("test_video_output/challenge_fit.mp4"))
Out[32]:
In [33]:
vp.save_to_file("test_videos/solidWhiteRight.mp4", "test_video_output/solidWhiteRight_smooth.mp4", vp.detect_smooth_lane_lines)
In [34]:
HTML("""
<video width="960" height="540" controls>
<source src="{0}">
</video>
""".format("test_video_output/solidWhiteRight_smooth.mp4"))
Out[34]:
In [35]:
vp.save_to_file("test_videos/solidYellowLeft.mp4", "test_video_output/solidYellowLeft_largest_smooth.mp4", vp.detect_smooth_lane_lines)
In [36]:
HTML("""
<video width="960" height="540" controls>
<source src="{0}">
</video>
""".format("test_video_output/solidYellowLeft_largest_smooth.mp4"))
Out[36]:
In [37]:
vp.save_to_file("test_videos/challenge.mp4", "test_video_output/challenge_largest_smooth.mp4", vp.detect_smooth_lane_lines)
In [38]:
HTML("""
<video width="960" height="540" controls>
<source src="{0}">
</video>
""".format("test_video_output/challenge_largest_smooth.mp4"))
Out[38]:
In [39]:
vp.save_to_file("test_videos/solidYellowLeft.mp4", "test_video_output/solidYellowLeft_centerline.mp4", vp.detect_centerline)
In [40]:
vp.save_to_file("test_videos/challenge.mp4", "test_video_output/challenge_centerline.mp4", vp.detect_centerline)
In [41]:
vp.save_to_file("test_videos/solidYellowLeft.mp4", "test_video_output/solidYellowLeft_zone_raw.mp4", vp.detect_lane_zone_raw)
In [42]:
vp.save_to_file("test_videos/challenge.mp4", "test_video_output/challenge_zone_raw.mp4", vp.detect_lane_zone_raw)
In [43]:
vp.save_to_file("test_videos/solidYellowLeft.mp4", "test_video_output/solidYellowLeft_zone.mp4", vp.detect_lane_zone_smooth)
In [44]:
vp.save_to_file("test_videos/challenge.mp4", "test_video_output/challenge_zone.mp4", vp.detect_lane_zone_smooth)
In [45]:
HTML("""
<video width="960" height="540" controls>
<source src="{0}">
</video>
""".format("test_video_output/solidYellowLeft_centerline.mp4"))
Out[45]:
In [46]:
HTML("""
<video width="960" height="540" controls>
<source src="{0}">
</video>
""".format("test_video_output/challenge_centerline.mp4"))
Out[46]:
In [47]:
HTML("""
<video width="960" height="540" controls>
<source src="{0}">
</video>
""".format("test_video_output/solidYellowLeft_zone_raw.mp4"))
Out[47]:
In [48]:
HTML("""
<video width="960" height="540" controls>
<source src="{0}">
</video>
""".format("test_video_output/challenge_zone_raw.mp4"))
Out[48]:
In [49]:
HTML("""
<video width="960" height="540" controls>
<source src="{0}">
</video>
""".format("test_video_output/solidYellowLeft_zone.mp4"))
Out[49]:
In [50]:
HTML("""
<video width="960" height="540" controls>
<source src="{0}">
</video>
""".format("test_video_output/challenge_zone.mp4"))
Out[50]: